feat: implement DuckDB filesystem integration for Vortex file handling by iceboundrock · Pull Request #6198 · vortex-data/vortex

iceboundrock · 2026-01-29T05:33:02Z

This PR addresses DuckDB extension to use DuckDB FileSystem #5767
This PR utilizes AI (codex from OpenAI) to generate code.

vortex-duckdb/src/scan.rs

codspeed-hq · 2026-01-29T09:48:19Z

Merging this PR will not alter performance

✅ 1135 untouched benchmarks
⏩ 1268 skipped benchmarks¹

_{Comparing Lychee-Technology:develop (b2db9de) with develop (6f10740)}

1268 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

0ax1

Thanks for taking the time to look into this. There's a couple of issues around thread safety / locking, error handling and memory leaks.

If you used AI to assist in writing the code for your PR please mention this in your PR description (see: https://github.com/vortex-data/vortex/blob/develop/CONTRIBUTING.md).

vortex-duckdb/src/copy.rs

vortex-duckdb/cpp/file_system.cpp

vortex-duckdb/src/duckdb/file_system.rs

vortex-duckdb/src/copy.rs

vortex-duckdb/src/utils/glob.rs

vortex-duckdb/src/duckdb/file_system.rs

vortex-duckdb/cpp/error.cpp

vortex-duckdb/src/duckdb/client_context.rs

vortex-duckdb/src/scan.rs

0ax1

Almost there, thanks for taking the time to address the comments!

vortex-duckdb/src/scan.rs

vortex-duckdb/cpp/include/duckdb_vx/file_system.h

iceboundrock · 2026-02-11T19:13:40Z

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

0ax1 · 2026-02-11T19:27:25Z

Almost there, thanks for taking the time to address the comments!

Hi @0ax1 , any other comments?

Currently swamped with other work but will get back to it when I find a free moment.

vortex-duckdb/src/scan.rs

gatesn · 2026-02-13T19:39:55Z

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

iceboundrock · 2026-02-13T21:16:55Z

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?

I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

Could you please elaborate on what config refers to? If it’s S3 credentials, HTTP Proxy configuration, etc., ~~I don’t believe we can read them from DuckDB. They are stored in Secrets Manager and the read API redacts sensitive information.~~

We probably can do the same as the httpfs extension does. Code link:

1
2

Signed-off-by: Ruoshi <me@ruoshi.li>

…chemas. Signed-off-by: Ruoshi <me@ruoshi.li>

0ax1 · 2026-02-16T11:20:19Z

@iceboundrock - quick question on this. Do you know if it's possible to grab the config back out from DuckDB's filesystems?
I don't want to block this PR, don't worry(!), but we do have plans to implement a high performance I/O layer at some point and it would probably make more sense to try and configure our own I/O using duckdb configs, rather than attempt to optimize calling back into DuckDB's I/O layer.

Could you please elaborate on what config refers to? If it’s S3 credentials, HTTP Proxy configuration, etc., ~~I don’t believe we can read them from DuckDB. They are stored in Secrets Manager and the read API redacts sensitive information.~~

We probably can do the same as the httpfs extension does. Code link:
* [1](https://github.com/duckdb/duckdb-httpfs/blob/6afe39d5be688b032146baa97e49601b50c9456e/src/httpfs_extension.cpp#L73-L105)

* [2](https://github.com/duckdb/duckdb-httpfs/blob/main/src/create_secret_functions.cpp)

Let's handle this in a follow up to limit the scope of this PR.

0ax1

PR looks great and CI (including all benchmarks) pass.

Last bit of fine-tuning and this is ready to land.

0ax1 · 2026-02-16T11:02:22Z

vortex-duckdb/cpp/file_system.cpp

+
+    try {
+        auto *wrapper = reinterpret_cast<FileHandleWrapper *>(handle);
+        wrapper->handle->Write(QueryContext(), const_cast<uint8_t *>(buffer), len, offset);


Drop the const_cast and adjust the signature accordingly.

0ax1 · 2026-02-16T11:07:59Z

vortex-duckdb/src/duckdb/client_context.rs

+    }
+}
+
+// SAFETY: ClientContext is an opaque pointer owned by DuckDB and remains valid for the lifetime of


These are actually not needed, as we pass around SendableClientContext.

Drop:

unsafe impl Send for ClientContext {} unsafe impl Sync for ClientContext {}

0ax1 · 2026-02-16T11:08:45Z

vortex-duckdb/src/duckdb/file_system.rs

+        let mut out_len: cpp::idx_t = 0;
+        let offset = self.pos;
+
+        let status = unsafe {


Use spawn blocking, instead of a blocking write.

0ax1 · 2026-02-16T11:12:53Z

vortex-duckdb/src/duckdb/file_system.rs

+    vortex_err!("{message}")
+}
+
+pub fn duckdb_fs_glob(ctx: &ClientContext, pattern: &str) -> VortexResult<Vec<url::Url>> {


pub(crate)

0ax1 · 2026-02-16T11:13:48Z

vortex-duckdb/src/duckdb/file_system.rs

+}
+
+/// A VortexReadAt implementation backed by DuckDB's filesystem (e.g., httpfs/s3).
+pub struct DuckDbFsReader {


pub(crate)

0ax1 · 2026-02-16T11:14:15Z

vortex-duckdb/src/duckdb/file_system.rs

+}
+
+impl DuckDbFsReader {
+    pub unsafe fn open_url(


pub(crate)

0ax1 · 2026-02-16T11:14:52Z

vortex-duckdb/src/duckdb/file_system.rs

+unsafe impl Send for DuckDbFsReader {}
+unsafe impl Sync for DuckDbFsReader {}
+
+pub struct DuckDbFsWriter {


pub(crate)

0ax1 · 2026-02-16T11:15:21Z

vortex-duckdb/src/duckdb/file_system.rs

+}
+
+impl DuckDbFsWriter {
+    pub unsafe fn create(ctx: cpp::duckdb_vx_client_context, path: &str) -> VortexResult<Self> {


pub(crate)

0ax1 · 2026-02-16T11:15:49Z

vortex-duckdb/src/duckdb/file_system.rs

+    Ok(urls)
+}
+
+pub unsafe fn duckdb_fs_create_writer(


pub(crate)

AdamGS reviewed Jan 29, 2026

View reviewed changes

vortex-duckdb/src/scan.rs Outdated Show resolved Hide resolved

joseph-isaacs added the action/benchmark Trigger full benchmarks to run on this PR label Jan 29, 2026

0ax1 self-requested a review January 29, 2026 10:40

0ax1 added action/benchmark-sql Trigger SQL benchmarks to run on this PR changelog/feature A new feature and removed action/benchmark Trigger full benchmarks to run on this PR labels Jan 29, 2026

joseph-isaacs added action/benchmark-sql Trigger SQL benchmarks to run on this PR and removed action/benchmark-sql Trigger SQL benchmarks to run on this PR labels Jan 29, 2026

0ax1 reviewed Jan 29, 2026

View reviewed changes

joseph-isaacs added action/benchmark-sql Trigger SQL benchmarks to run on this PR and removed action/benchmark-sql Trigger SQL benchmarks to run on this PR labels Feb 2, 2026

0ax1 reviewed Feb 2, 2026

View reviewed changes

0ax1 reviewed Feb 4, 2026

View reviewed changes

vortex-duckdb/cpp/error.cpp Outdated Show resolved Hide resolved

0ax1 reviewed Feb 4, 2026

View reviewed changes

vortex-duckdb/src/duckdb/client_context.rs Show resolved Hide resolved

0ax1 reviewed Feb 4, 2026

View reviewed changes

vortex-duckdb/src/scan.rs Outdated Show resolved Hide resolved

0ax1 reviewed Feb 4, 2026

View reviewed changes

0ax1 reviewed Feb 5, 2026

View reviewed changes

vortex-duckdb/src/scan.rs Outdated Show resolved Hide resolved

0ax1 reviewed Feb 5, 2026

View reviewed changes

vortex-duckdb/cpp/include/duckdb_vx/file_system.h Outdated Show resolved Hide resolved

ruoshili force-pushed the develop branch 2 times, most recently from 66fedc0 to b6e15a3 Compare February 11, 2026 02:21

gatesn reviewed Feb 13, 2026

View reviewed changes

vortex-duckdb/src/scan.rs Outdated Show resolved Hide resolved

gatesn added action/benchmark-sql Trigger SQL benchmarks to run on this PR and removed action/benchmark-sql Trigger SQL benchmarks to run on this PR labels Feb 13, 2026

iceboundrock added 2 commits February 14, 2026 16:16

feat: implement DuckDB filesystem integration for Vortex file handling

dc7a4ae

Signed-off-by: Ruoshi <me@ruoshi.li>

refactor: simplify open_file function by letting duckdb handles all s…

b2db9de

…chemas. Signed-off-by: Ruoshi <me@ruoshi.li>

iceboundrock force-pushed the develop branch from 20ea783 to b2db9de Compare February 15, 2026 00:18

0ax1 added action/benchmark-sql Trigger SQL benchmarks to run on this PR and removed action/benchmark-sql Trigger SQL benchmarks to run on this PR labels Feb 16, 2026

0ax1 reviewed Feb 16, 2026

View reviewed changes

Conversation

iceboundrock commented Jan 29, 2026 • edited by 0ax1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codspeed-hq bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

0ax1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

0ax1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

iceboundrock commented Feb 11, 2026

Uh oh!

0ax1 commented Feb 11, 2026

Uh oh!

Uh oh!

gatesn commented Feb 13, 2026

Uh oh!

iceboundrock commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0ax1 commented Feb 16, 2026

Uh oh!

0ax1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

iceboundrock commented Jan 29, 2026 •

edited by 0ax1

Loading

codspeed-hq bot commented Jan 29, 2026 •

edited

Loading

iceboundrock commented Feb 13, 2026 •

edited

Loading